Vehicle exterior environment recognition apparatus

ABSTRACT

A vehicle exterior environment recognition apparatus includes a road surface identifying unit, a three-dimensional object identifying unit, a road surface determining unit, and a three-dimensional object composition unit. The road surface identifying unit identifies a road surface in an image. The three-dimensional object identifying unit identifies three-dimensional objects each having a height extending vertically upward from the identified road surface. When the identified three-dimensional objects are separated and are located at respective positions distant from an own vehicle by a same relative distance, the road surface determining unit performs a determination of whether a three-dimensional-object-intervening region between the identified three-dimensional objects has a correspondence to the road surface. When the three-dimensional-object-intervening region is determined to have no correspondence to the road surface, the three-dimensional object composition unit regards the identified three-dimensional objects separated from each other as candidate parts of a unified three-dimensional object.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority from Japanese Patent ApplicationNo. 2017-248484 filed on Dec. 25, 2017, the entire contents of which arehereby incorporated by reference.

BACKGROUND

The technology relates to a vehicle exterior environment recognitionapparatus that identifies a specific object present in a travelingdirection of an own vehicle.

A technique has been proposed which detects a three-dimensional object,such as a preceding vehicle located ahead of an own vehicle, to performcontrol that avoids contact with the preceding vehicle (i.e., contactavoidance control) or to perform control that keeps a predeterminedinter-vehicular distance from the own vehicle to the preceding vehicle(i.e., cruise control). For example, reference is made to JapanesePatent No. 3349060.

SUMMARY

An aspect of the technology provides a vehicle exterior environmentrecognition apparatus configured to recognize an environment outside anown vehicle. The apparatus includes: a road surface identifying unitconfigured to identify a road surface in an image; a three-dimensionalobject identifying unit configured to identify three-dimensional objectseach having a height extending vertically upward from the identifiedroad surface; a road surface determining unit configured to perform adetermination of whether a three-dimensional-object-intervening regionbetween the three-dimensional objects identified by thethree-dimensional object identifying unit has a correspondence to theroad surface, when the identified three-dimensional objects areseparated from each other and are located at respective positionsdistant from the own vehicle by a same relative distance; and athree-dimensional object composition unit configured to regard theidentified three-dimensional objects that are separated from each otheras candidate parts of a unified three-dimensional object, when thethree-dimensional-object-intervening region is determined to have nocorrespondence to the road surface by the road surface determining unit.

An aspect of the technology provides a vehicle exterior environmentrecognition apparatus configured to recognize an environment outside anown vehicle. The apparatus includes circuitry configured to identify aroad surface in an image, identify three-dimensional objects each havinga height extending vertically upward from the identified road surface,perform a determination of whether athree-dimensional-object-intervening region between the identifiedthree-dimensional objects has a correspondence to the road surface, whenthe identified three-dimensional objects are separated from each otherand are located at respective positions distant from the own vehicle bya same relative, and regard the identified three-dimensional objectsthat are separated from each other as candidate parts of a unifiedthree-dimensional object, when the three-dimensional-object-interveningregion is determined to have no correspondence to the road surface.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are included to provide a furtherunderstanding of the technology and are incorporated in and constitute apart of this specification. The drawings illustrate exampleimplementations and, together with the specification, serve to explainthe principles of the technology.

FIG. 1 is a block diagram illustrating an example relation of connectionin a vehicle exterior environment recognition system according to oneimplementation of the technology.

FIG. 2A is a diagram illustrating an example luminance image, and FIG.2B is a diagram illustrating an example distance image.

FIG. 3 is a block diagram illustrating a schematic configuration of avehicle exterior environment recognition apparatus according to oneimplementation of the technology.

FIG. 4 is a flowchart illustrating an example vehicle exteriorenvironment recognition procedure.

FIGS. 5A and 5B are diagrams illustrating an example process ofidentifying a road surface region.

FIG. 6 is a diagram illustrating an example of a histogram.

FIGS. 7A to 7C are diagrams illustrating an example of a road surfacemodel.

FIG. 8 is a diagram illustrating an example process of identifying athree-dimensional object performed by a road surface identifying unit.

FIG. 9 is a diagram illustrating an example of a process performed onblocks corresponding to the road surface.

FIG. 10 is a diagram illustrating an example integration processperformed by the road surface identifying unit.

FIGS. 11A and 11B are diagrams illustrating an example process performedby a road surface determining unit.

FIG. 12 is a diagram illustrating an example process performed by theroad surface determining unit.

DETAILED DESCRIPTION

In the following, some implementations of the technology are describedin detail with reference to the accompanying drawings. Note that sizes,materials, specific values, and any other factors illustrated inrespective implementations are illustrative for easier understanding ofthe technology, and are not intended to limit the scope of thetechnology unless otherwise specifically stated. Further, elements inthe following example implementations which are not recited in amost-generic independent claim of the technology are optional and may beprovided on an as-needed basis. Throughout the present specification andthe drawings, elements having substantially the same function andconfiguration are denoted with the same reference numerals to avoid anyredundant description. Further, elements that are not directly relatedto the technology are unillustrated in the drawings. The drawings areschematic and are not intended to be drawn to scale.

Examples of a three-dimensional object present in a traveling directionof an own vehicle may include a preceding vehicle traveling in the samedirection as the own vehicle and a pedestrian moving in a lateraldirection across the traveling path of the own vehicle. For example,grouping may be performed of a plurality of blocks that are located atrespective positions distant from the own vehicle by the same relativedistance in a captured image of an environment ahead of the own vehicle.When the external appearance of a three-dimensional object representedby the resultant group has a feature corresponding to a rear surface ofa vehicle, the three-dimensional object may be identified as a precedingvehicle.

However, in a case where a relative distance from the own vehicle to amiddle portion of the rear surface of the preceding vehicle is difficultto be obtained due to a factor, such as backlight, and only a relativedistance from the own vehicle to two opposite side portions of thepreceding vehicle that are separated from each other is obtained, thetwo opposite side portions that should be identified as a unifiedpreceding vehicle can be misidentified as two differentthree-dimensional objects that are separated from each other.

One possible measure to identify the two opposite side portionsseparated from each other as a unified object may include relaxation ofa threshold range of relative distance or a threshold for identificationof a unified object. However, such simple relaxation of the thresholdvalue can cause misidentification, as a unified three-dimensionalobject, of three-dimensional objects, such as road cones, that areseparated from each other and aligned at respective positions distantfrom the own vehicle by the same relative distance. This can cause anexcess contact avoidance control. In such a case, only the presence ofan additional three-dimensional object disposed between the twodifferent three-dimensional objects may help to prevent or inhibit themisidentification of these three-dimensional objects as a unifiedthree-dimensional object. However, in a case of no additionalthree-dimensional object presents between two three-dimensional objects,it is difficult to properly extract a three-dimensional object thatshould be identified as a unified preceding vehicle.

It is desirable to provide a vehicle exterior environment recognitionapparatus that achieves proper extraction of a three-dimensional object.

[Vehicle Exterior Environment Recognition System 100]

FIG. 1 is a block diagram illustrating an example relation of connectionin a vehicle exterior environment recognition system 100 according to anexample implementation of the technology. The vehicle exteriorenvironment recognition system 100 may include at least oneimage-capturing unit 110, a vehicle exterior environment recognitionapparatus 120, and a vehicle controller (e.g., an engine control unit(ECU)) 130. The number of the image-capturing units 110 may be, forexample but not limited to, two, in the example implementation.

The two image-capturing units 110 each may include an imaging devicesuch as, but not limited to, a charge-coupled device (CCD) and acomplementary metal-oxide semiconductor (CMOS). The image-capturingunits 110 each may capture an image of an environment outside and aheadof an own vehicle 1 (i.e., vehicle exterior environment), and maygenerate a luminance image (e.g., color image or monochrome image)including at least luminance information. The two image-capturing units110 may be disposed separated away from each other in a substantiallyhorizontal direction. The two image-capturing units 110 may be disposedso that their respective optical axes are to be substantially parallelto each other along a traveling direction of the own vehicle 1. Theimage-capturing units 110 may continuously generate a luminance imagefor each frame of, for example but not limited to, 1/60 second (at aframe rate of 60 fps). The luminance image may be obtained as a resultof the image capturing performed on a detection region ahead of the ownvehicle 1. Non-limiting examples of a three-dimensional object to berecognized with the image-capturing units 110 may include athree-dimensional object independently present, as well as an objectidentifiable as a part of the three-dimensional object. Non-limitingexamples of the independently-present three-dimensional object mayinclude a bicycle, a pedestrian, a vehicle, a traffic light, a roadsign, a guardrail, and a building. Non-limiting examples of the objectidentifiable as a part of the three-dimensional object may include awheel of a bicycle.

The vehicle exterior environment recognition apparatus 120 may receivethe respective luminance images from the two image-capturing units 110,and perform a so-called pattern matching between the luminance images.The pattern matching may involve extracting any block from one of theluminance images and searching the other luminance image for a blockcorresponding to the extracted block. The block may be, for example butnot limited to, an array of four horizontal pixels and four verticalpixels. Through the pattern matching, the vehicle exterior environmentrecognition apparatus 120 may derive parallax information indicating aparallax and a position of any block in the luminance image. As usedherein, the term “horizontal” refers to a lateral direction of a screenof the captured image, and the term “vertical” refers to a longitudinaldirection of the screen of the captured image. In the example patternmatching, luminance (Y) may be compared per block between the twoluminance images. Non-limiting examples of a scheme for the comparisonmay include SAD (Sum of Absolute Difference) that obtains luminancedifferences, SSID (Sum of Squared Intensity Difference) that uses thesquared differences, and ZNCC (Zero-mean Normalized Cross Correlation)that obtains similarity of variance calculated by subtracting an averageluminance value from a luminance value of each pixel. The vehicleexterior environment recognition apparatus 120 may perform the parallaxderiving process for all blocks in the detection region, on a blockbasis. The detection region may be, for example but not limited to, anarray of 600 horizontal pixels by 200 vertical pixels. In this exampleimplementation, each block may include the array of four horizontalpixels by four vertical pixels; however, any number of pixels may beincluded in each block.

The vehicle exterior environment recognition apparatus 120 may derivethe parallax per block, i.e., on a detection resolution basis. However,the vehicle exterior environment recognition apparatus 120 may havedifficulties in recognizing which part of the three-dimensional objectthe block belongs to and which type the three-dimensional object is.Hence, the parallax information may be derived independently on thedetection resolution basis (e.g., on the block basis) with respect tothe detection region, not on a three-dimensional object basis. An imagein association with the derived parallax information is hereinafterreferred to as a “distance image”, for discrimination from the luminanceimage describe above.

FIG. 2A illustrates an example of the luminance image (luminance image126), and FIG. 2B illustrates an example of the distance image (distanceimage 128). For example, the two image-capturing units 110 may generatethe respective luminance images 126 of a detection region 124, asillustrated in FIG. 2A. Note that only one of the luminance images 126is schematically illustrated in FIG. 2A for easier understanding of theexample implementation of the technology. The vehicle exteriorenvironment recognition apparatus 120 may obtain a parallax per blockfrom the luminance images 126 and generate the distance image 128illustrated in FIG. 2B. Each block in the distance image 128 may beassociated with the corresponding parallax. For convenience ofillustration, the blocks in association with the respective parallaxesare each represented by a solid dot, in FIG. 2B.

The vehicle exterior environment recognition apparatus 120 may alsoidentify a road surface using three-dimensional position information ina real space. The three-dimensional position information may include aluminance value (e.g., color value) calculated from the luminance image126 and a relative distance from the own vehicle 1 calculated from thedistance image 128. Thereafter, the vehicle exterior environmentrecognition apparatus 120 may perform grouping of blocks that arelocated on the identified road surface, equal to each other in colorvalues, and close to each other in the three-dimensional positioninformation, into a single three-dimensional object. Thereafter, thevehicle exterior environment recognition apparatus 120 may identify towhich object (e.g., the preceding vehicle or bicycle) thethree-dimensional object in the detection region ahead of the ownvehicle 1 corresponds. After the identification of the three-dimensionalobject, the vehicle exterior environment recognition apparatus 120 mayperform control of the own vehicle 1 to avoid contact with thethree-dimensional object (i.e., contact avoidance control) or control ofthe own vehicle 1 to keep a predetermined inter-vehicular distance fromthe preceding vehicle (i.e., cruise control) for safety. The relativedistance may be determined by converting the parallax information perblock of the distance image 128 into the three-dimensional positioninformation by a so-called stereo method. The stereo method may derive,from the parallax of any part of the three-dimensional object, therelative distance between the part of the three-dimensional object andthe image-capturing units 110 by a triangulation method.

The vehicle controller 130 may control the own vehicle 1 by receivinginformation on an operation input of the driver through a steering wheel132, an accelerator pedal 134, and a brake pedal 136 and sending theinformation to a steering mechanism 142, a drive mechanism 144, and abrake mechanism 146. The vehicle controller 130 may also control thesteering mechanism 142, the drive mechanism 144, and the brake mechanism146, in accordance with instructions from the vehicle exteriorenvironment recognition apparatus 120.

As described above, the vehicle exterior environment recognition system100 may perform the grouping of blocks that are equal to each other incolor values and close to each other in the three-dimensional positioninformation, into a single three-dimensional object. For example, when apreceding vehicle presents ahead of the own vehicle 1, a plurality ofblocks that correspond to a rear surface of the preceding vehicle andthus are equal to each other in relative distance may be grouped into athree-dimensional object. The three-dimensional object may be identifiedas a preceding vehicle depending on its feature.

However, in a case where a relative distance from the own vehicle 1 to amiddle portion of the rear surface of the preceding vehicle is difficultto be obtained due to a factor, such as backlight, and only a relativedistance from the own vehicle 1 to two opposite side portions of thepreceding vehicle that are separated from each other is obtained, thetwo opposite side portions that should be identified as a unifiedpreceding vehicle can be misidentified as two differentthree-dimensional objects that are separated from each other. Onepossible measure to identify the two opposite side portions separatedfrom each other as a unified object may include simple relaxation of athreshold range of relative distance or a threshold value foridentification of a unified three-dimensional object. However, suchsimple relaxation of the threshold value can cause misidentification, asa unified three-dimensional object, of three-dimensional objects, suchas road cones, that are separated from each other and aligned atrespective positions distant from the own vehicle 1 by the same relativedistance. Accordingly, an object of at least one implementation of thetechnology is to achieve proper extraction of a three-dimensional objectthat should be identified as a unified preceding vehicle by utilizing arelation between the three-dimensional object and the road surface.

In the following, a description is given in detail of a configuration ofthe vehicle exterior environment recognition apparatus 120 that achievesthe example object of at least one implementation of the technology.Given here is a detailed description of an example process ofidentifying a road surface and a three-dimensional object in thedetection region ahead of the own vehicle 1, which is one feature ofthis example implementation. Note that a configuration less related tofeatures of the implementation will not be described in detail.

[Vehicle Exterior Environment Recognition Apparatus 120]

FIG. 3 is a block diagram illustrating a schematic configuration of thevehicle exterior environment recognition apparatus 120 according to anexample implementation of the technology. As illustrated in FIG. 3, thevehicle exterior environment recognition apparatus 120 may include aninterface 150, a data storage 152, and a central controller 154.

The interface 150 may exchange information bidirectionally betweendevices including, without limitation, the image-capturing unit 110 andthe vehicle controller 130. The data storage 152 may include a randomaccess memory (RAM), a flash memory, a hard disk drive (HDD), or anyother suitable storage device. The data storage 152 may store variouspieces of information necessary for processes to be carried out bycomponents described hereinafter.

The central controller 154 may include a semiconductor integratedcircuit, and may control devices including, without limitation, theinterface 150 and the data storage 152 through a system bus 156. Thesemiconductor integrated circuit may have devices such as, but notlimited to, a central processing circuit (CPU), a read only memory (ROM)in which programs, etc., are stored, and a random access memory (RAM)serving as a work area. In this example implementation, the centralcontroller 154 may also serve as a a road surface identifying unit 160,a three-dimensional object identifying unit 162, a road surfacedetermining unit 164, and a three-dimensional object composition unit166. In the following, a vehicle exterior environment recognitionprocedure that identifies a road surface is described in detail as onefeature of this example implementation, with reference to the operationof each of the components of the central controller 154.

[Vehicle Exterior Environment Recognition Procedure]

FIG. 4 is a flowchart of an example of the vehicle exterior environmentrecognition procedure. In the vehicle exterior environment recognitionprocedure, the road surface identifying unit 160 performs a road surfaceidentifying process (S200) of identifying a road surface in the image.Thereafter, the three-dimensional object identifying unit 162 performs athree-dimensional object identifying process (S202) of identifying athree-dimensional object on the basis of the identified road surface.Thereafter, the road surface determining unit 164 performs a roadsurface determining process (S204) of determining whether a regionbetween three-dimensional objects that are separated from each other anddistant from the own vehicle 1 by the same relative distance has acorrespondence to the road surface. When the road surface determiningunit 164 determines that the region between the three-dimensionalobjects has the correspondence to the road surface (YES in S206), thevehicle exterior environment recognition procedure may be terminated.When the road surface determining unit 164 determines that the regionbetween the three-dimensional objects has no correspondence to the roadsurface (NO in S206), the three-dimensional object composition unit 166performs a three-dimensional object composition process (S208) ofregarding the three-dimensional objects that are separated from eachother as candidate parts of a unified three-dimensional object. In oneimplementation of the technology, the region between thethree-dimensional objects may serve as a“three-dimensional-object-intervening region”.

[Road Surface Identifying Process S200]

The road surface identifying unit 160 identifies a road surface regionthat corresponds to the road surface in the luminance image 126 or thedistance image 128. For example, firstly, the road surface identifyingunit 160 identifies the road surface region on the basis of right andleft lane lines (e.g., white lines) of a lane on which the own vehicle 1is traveling.

FIGS. 5A and 5B illustrate an example of the road surface identifyingprocess. The road surface may be provided with lane lines for smoothtraveling of a vehicle. In the example illustrated in FIG. 5A, forexample, the road 200 in the luminance image 126 may be divided into twolanes 202 a and 202 b by a total of three lane lines 210 a, 210 b, and210 c. The lane line 210 b may be provided in the middle of the road 200along the horizontal direction. The lane line 210 b may be hereinafterreferred to as middle lane line 210 b. The lane lines 210 a and 210 cmay be provided on respective ends of the road 200. The lane lines 210 amay be hereinafter referred to as a left lane line 210 a, and the lanelines 210 c may be hereinafter referred to as a right lane line 210 c.

Referring to FIG. 5B, the lane 202 a on which the own vehicle 1 istraveling may be defined between the left lane line 210 a and the middlelane line 210 b. In other words, the left lane line 210 a may beprovided on a leftmost side of the lane 202 a, and the middle lane line210 b may be provided on a rightmost side of the lane line 202 a. Theroad surface identifying unit 160 may set an imaginary left limit line212 a that is shifted leftward by a predetermined distance (e.g., 10 cm)from the left lane line 210 a, as illustrated by a dashed line in FIG.5B. Likewise, the road surface identifying unit 160 may set an imaginaryright limit line 212 b that is shifted rightward by a predetermineddistance (e.g., 10 cm) from the middle lane line 210 b, as illustratedby another dashed line in FIG. 5B. Thereafter, the road surfaceidentifying unit 160 sets a region extending rightward from the leftlimit line 212 a and leftward from the right limit line 212 b to be aroad surface region 214. In other words, the road surface identifyingunit 160 sets a region extending between the left limit line 212 a andthe right limit line 212 b along the horizontal direction (i.e., ahatched region in FIG. 5B) to be the road surface region 214.

In this example implementation, the road surface region may beidentified only on the basis of the lane lines on the road surface, butthe identification of the road surface region should not be limited tothe example implementation. In another example implementation of thetechnology, the road surface region may be identified on the basis of afuture traveling path on which the own vehicle 1 is predicted to travelat a current steering angle, a current turning angle rate (i.e., yawrate), and a current speed, in addition to the lane lines on the roadsurface. For example, a road surface region that horizontally extends in2.5 meters or less in either side from a curve line indicating thefuture traveling path ahead of the own vehicle 1 may be identified asthe road surface region. Such complementary identification improvesaccuracy in identifying the road surface region.

Thereafter, the road surface identifying unit 160 may extract, from theroad surface region 214 in the distance image 128, all of the blockshaving the respective relative distances (i.e., all of the blocks ofwhich relative distances are obtained through the pattern matching). Theroad surface identifying unit 160 may generate a road surface model onthe basis of the extracted blocks. The generation of the road surfacemodel will now be described.

FIG. 6 illustrates generation of an example histogram, and FIGS. 7A to7C each illustrate an example of the road surface model. First, the roadsurface identifying unit 160 may generate a histogram of the relativedistances of any horizontal array of blocks along the horizontaldirection in the road surface region 214. For example, the road surfaceidentifying unit 160 may poll the relative distances of any blockslocated at the same vertical position in the road surface region 214 inthe image. In other words, the road surface identifying unit 160 maypoll the relative distances of all blocks in any horizontal array (e.g.,a cross-hatched array in (a) of FIG. 6). On the basis of the polledrelative distances, a histogram regarding any vertical position may begenerated, as illustrated in (b) of FIG. 6.

A hatched bar in the histogram indicates a relative distance having amaximum number of polls. The road surface identifying unit 160 may setthe relative distance indicated by the hatched bar to be arepresentative distance of the vertical position. The road surfaceidentifying unit 160 may repeat such a process for another verticalposition while changing a vertical position of interest in the roadsurface region 214, to derive the representative distances of respectivevertical positions in the road surface region 214.

For example, the road surface model 216 may be represented by a relationbetween a vertical axis indicating the vertical position in the distanceimage 128 and a horizontal axis indicating the relative distance in thedistance image 128. The road surface identifying unit 160 may plot, atthe respective vertical positions, the representative distances of therespective vertical positions (i.e., the representative distances of therespective horizontal arrays of blocks) to generate point groupsillustrated in FIG. 7A. From the point groups, the road surfaceidentifying unit 160 may generate an approximate straight line, which isillustrated by a solid line in FIG. 7A, by a least squares method, forexample. The road surface identifying unit 160 may set the approximatestraight line to be the road surface model 216. The road surface model216 helps to identify a varying feature, such as gradient, of the roadsurface. As used herein, the term “varying feature” refers to how theroad surface varies. In this example implementation, the road surfacemodel 216 may be derived in the form of the approximate straight linefor the purpose of illustration; however, the road surface model 216 maybe derived in the form of a multi-dimensional approximated curve, in analternative implementation of the technology.

The approximate straight line generated simply by the least squaresmethod, however, can include noise representative distances. The noiserepresentative distances may include error representative distances thatare obtained by erroneous pattern matching in the road surface region214, for example. Inclusion of such noise representative distances,which should be practically excluded, can cause deviation of theresultant approximate straight line from an appropriate position orinclination, as illustrated in FIG. 7A. This deviation can lead toimproper determination of a road having a gradient to be athree-dimensional object, or a failure in extracting a three-dimensionalobject actually present.

To address such a concern, in an example implementation of thetechnology, the Hough transform may be performed which detects astraight line in an image. Through the Hough transform, only a pointgroup of representative distances that form a common straight line, oronly a point group of representative distances that form no commonstraight line but form straight lines parallel to and close to eachother (i.e., straight lines apart from each other by a distance within apredetermined range) may be remained, whereas the other point groups ofrepresentative distances (e.g., point groups each surrounded by a dashedline in FIG. 7A) may be excluded as the noise representative distances.The road surface identifying unit 160 may perform the least squaresmethod only on the remaining point group of representative distances toderive an approximate straight line from the point group. Through theseprocesses, the road surface model 216 may be generated in the form ofthe appropriate approximate straight line on the basis of the properrepresentative distances, as illustrated in FIG. 7B. Since the Houghtransform is a currently-available technique for deriving a commonstraight line passing through a plurality of points, the descriptionthereof is not described in detail herein.

In this example implementation, the varying feature of the road surfacemay be represented in the form of the approximate straight line. Theroad in the detection region, however, does not necessarily vary in thesame fashion. For example, the road in the detection region may have asteep inclination, in some cases. In such cases, the Hough transform canexclude most of representative distances corresponding to a region ofthe road surface following the steep inclination, as the noiserepresentative distances.

To address such a concern, an example implementation of the technologymay derive another approximate straight line from the point group of therepresentative distances excluded by the Hough transform, in a casewhere the number of the representative distances excluded by the Houghtransform is equal to or greater than a predetermined number and havecontinuity to each other. It is indisputable that noise representativedistances may be excluded also in the generation of the otherapproximate straight line. Through these processes, the road surfacemodel 216 may be generated in the form of two approximate straight linescontinuous to each other, as illustrated in FIG. 7C.

Alternatively, the two approximate straight lines derived through theprocesses described above may intersect with each other at apredetermined angle, or may be coupled to each other with a transitioncurve having a predetermined radius. Additionally, although the twoapproximate straight lines may be generated in the foregoing exampleimplementations, three or more approximate straight lines may begenerated for a road with lots of variations.

[Three-Dimensional Object Identifying Process S202]

The three-dimensional object identifying unit 162 may identify athree-dimensional object having a height extending vertically upwardfrom the road surface on the basis of the road surface model 216generated as described above.

FIG. 8 is a diagram illustrating an example process of identifying athree-dimensional object performed by the three-dimensional objectidentifying unit 162. In FIG. 8, the road surface model 216 of the roadsurface region 214 at any timing is represented in the form of astraight line, and the blocks extending over the entire image and havingthe respective relative distances are plotted at the respective relativedistances and the respective vertical positions.

The three-dimensional object identifying unit 162 may compare therelative distance of each of the blocks with the road surface model 216to determine whether the block corresponds to part of athree-dimensional object. For example, the three-dimensional objectidentifying unit 162 may determine a block located within a road surfacerange 220 to correspond to part of the road surface. The road surfacerange 220 may be a region defined between an upper limit and a lowerlimit that are respectively indicated by dashed lines in FIG. 8. Theupper limit may be higher than the road surface model 216 by apredetermined distance (e.g., about 30 cm), and the lower limit may belower than the road surface model 216 by a predetermined distance (e.g.,about 30 cm).

On the other hand, the three-dimensional object identifying unit 162 mayregard a block located outside the road surface range 220 and above theupper limit of the road surface range 220 as a candidate part of athree-dimensional object, because the block protrudes upward from theroad surface. The three-dimensional object identifying unit 162 maythereafter perform grouping of the blocks that are regarded as thecandidate parts of a three-dimensional object having a height extendingvertically upward from the road surface and that are equal to each otherin the relative distance, into one group 222, which is surrounded by anoval in FIG. 8. The three-dimensional object identifying unit 162 mayidentify the group 222 to be a three-dimensional object. There arevarious currently-available technologies for determiningthree-dimensionality (e.g., shape, size, etc.) of the point groupidentified as a three-dimensional object and for determining which partof the three-dimensional object the point group belongs to; therefore,the description of the technologies is not described in detail herein.

Note that the three-dimensional object identifying unit 162 may performextraction of blocks that correspond to the road surface in parallel tothe extraction of the blocks that correspond to the three-dimensionalobject. The blocks that correspond to the road surface may be used inthe three-dimensional object composition process S208 described below.

FIG. 9 illustrates an example process performed on blocks correspondingto the road surface. In this example implementation, an array of fivehorizontal blocks by seven vertical blocks may be extracted from alower-left part of the luminance image 126 or the distance image 128illustrated in (a) of FIG. 9, for convenience of illustration.

For example, in a case where a block in the extracted array of blockshas a relative distance that is located within the road surface range220 (i.e., has a height extending in either vertical direction from theroad surface model 216 by a predetermined distance (e.g., 30 cm)), thethree-dimensional object identifying unit 162 determines that the blockhas a correspondence to the road surface, and may assign “1” to theblock.

In contrast, in a case where a block in the extracted array of blockshas a relative distance that is located outside the road surface range220, the three-dimensional object identifying unit 162 determines thatthe block has no correspondence to the road surface, and may assign “0(zero)” to the block. In a case where a block in the extracted array ofblocks has no relative distance, the three-dimensional objectidentifying unit 162 determines that the block has no correspondence tothe road surface, and may assign “0” to the block. In this way, each ofthe blocks in the image may be assigned with “1” or “0”, as illustratedin (b) of FIG. 9. The determination of whether a block has acorrespondence to the road surface is hereinafter also referred to as“determination of correspondence to the road surface”.

Thereafter, the three-dimensional object identifying unit 162 mayperform integration of the results (i.e., determination values) of thedetermination of correspondence to the road surface, on a block basis,along both or one of the vertical direction and the horizontaldirection.

FIG. 10 illustrates an example of the integration process performed bythe three-dimensional object identifying unit 162. Also in this exampleimplementation, the array of five horizontal blocks by seven verticalblocks may be extracted from the lower-left part of the luminance image126 or the distance image 128 illustrated in (a) of FIG. 9, forconvenience of illustration.

The three-dimensional object identifying unit 162 may performintegration of the determination values assigned to all of the blocksthat are located leftward and downward from any predetermined block. Thedetermination values may be obtained through the determination ofcorrespondence to the road surface, and may be each represented as “1”or “0” as described above. For example, referring to FIG. (b) of 10, anintegrated value of a predetermined block located at a horizontalposition 3 and a vertical position 4 may be an integrated value of atotal of twelve blocks that includes the predetermined block and all theblocks located leftward and downward from the predetermined block (i.e.,the blocks located at horizontal positions 1 to 3 and vertical positions1 to 4). In this example implementation, five blocks out of the twelveblocks are each assigned with “1” as illustrated in (a) of FIG. 10;therefore, the integrated value of the predetermined block located atthe horizontal position 3 and the vertical position 4 is “5”.

Note that the integrated value of any predetermined block is readilycalculated using blocks adjacent to the predetermined block. Forexample, referring to (c) of FIG. 10, an integrated value “ai” of thepredetermined block may be calculated by the following expression:a+bi+ci−di

where “a” represents the determination value of the predetermined blockthat is obtained through the determination of correspondence to the roadsurface, “bi” represents an integrated value of a block at the immediateleft of the predetermined block, “ci” represents an integrated value ofa block immediately below the predetermined block, and “di” representsan integrated value of a block at the immediate lower left of thepredetermined block. Note that, when no corresponding block exists in aleft-end part or a bottom-end part of the image, “0” may be assigned toa deemed block.

For example, the integrated value of the predetermined block at thehorizontal position 3 and the vertical position 4 illustrated in (b) ofFIG. 10 is “5”, which is calculated by the expression 0+4+3−2, where “0”is the determination value of the predetermined block obtained throughthe determination of correspondence to the road surface, “4” is theintegrated value of the block at the immediate left of the predeterminedblock, “3” is the integrated value of the block immediately below thepredetermined block, and “2” is the integrated value of the block at theimmediate lower left of the predetermined block. In this way, anintegrated value map 224 illustrated in (b) of FIG. 10, for example, maybe generated.

Note that the three-dimensional object identifying unit 162 may performthe determination of correspondence to the road surface illustrated in(a) of FIG. 10 and the integration of the determination valuesillustrated in (b) of FIG. 10 in parallel to each other. In an exampleimplementation of the technology, the three-dimensional objectidentifying unit 162 may perform the determination of correspondence tothe road surface for each predetermined block from the lower-left partof the image in a rightward direction along the horizontal side of theimage, while calculating the integrated value of the predetermined blockusing the integrated values of blocks adjacent to the predeterminedblock that are based on the determination values. After reaching a blockat an end point of the horizontal side (i.e., a right end) of the image,the three-dimensional object identifying unit 162 may perform thedetermination of correspondence to the road surface for each blocklocated at a vertical position immediately above the vertical positionof the processed blocks, from a starting point of the horizontal side(i.e., a left end) of the image in the rightward direction along thehorizontal side of the image, while calculating the integrated value ofthe block using the integrated values of blocks adjacent to the blockthat are based on the determination values.

Such parallel execution of the determination of correspondence to theroad surface and the integration of the determination values helps toreduce processing load.

[Road Surface Determining Process S204]

FIGS. 11A and 11B illustrate an example process performed by the roadsurface determining unit 164. The luminance image 126 illustrated inFIG. 11A may include an image of a preceding vehicle 230. In thisexample implementation, however, a relative distance from the ownvehicle 1 to a middle portion 230 a of the rear surface of the precedingvehicle 230 is difficult to be obtained due to a factor, such asbacklight, and only a relative distance from the own vehicle 1 to theopposite side portions 230 b of the preceding vehicle 230 that areseparated from each other is obtained. Accordingly, the two oppositeside portions 230 b that should be identified as a unified precedingvehicle 230 can be misidentified as two different three-dimensionalobjects that are separated from each other.

The luminance image 126 illustrated in FIG. 11B may include an image ofa plurality of three-dimensional objects 232. In this exampleimplementation, the three-dimensional objects 232 are four road cones,and a relative distance from the own vehicle 1 to each of thethree-dimensional objects 232 is properly obtained. Of the fourthree-dimensional objects 232, two three-dimensional objects 232 locatedmore adjacent to the own vehicle 1 than the other two three-dimensionalobjects 232 are equal to each other in the relative distance.Accordingly, in both the example illustrated in FIG. 11A and the exampleillustrated in FIG. 11B, the two different three-dimensional objects areseparated from each other and are equal to each other in the relativedistance; however, the two different three-dimensional objects (oppositeside portions 230 b) should be identified as the unified precedingvehicle 230 in the example illustrated in FIG. 11B, whereas the twodifferent three-dimensional objects 232 should be identified as twodifferent three-dimensional objects in the example illustrated in FIG.11B.

It is found through the comparison between FIG. 11A and FIG. 11B that anobject to which blocks in a three-dimensional-object-intervening regionbetween the three-dimensional objects of FIG. H A correspond isdifferent from an object to which blocks in athree-dimensional-object-intervening region between thethree-dimensional objects of FIG. 11B correspond. For example, theobject to which the blocks in the three-dimensional-object-interveningregion of FIG. 11A correspond may be the middle portion 230 a of thepreceding vehicle 230. The object is thus unlikely to be identified asthe road surface. In contrast, the object to which the blocks in thethree-dimensional-object-intervening region of FIG. 11B correspond maybe the road surface 234. The object is thus likely to be identified asthe road surface.

Accordingly, when the three-dimensional objects are separated from eachother and are distant from the own vehicle 1 by the same relativedistance, the road surface determining unit 164 determines whether thethree-dimensional-object-intervening region has a correspondence to theroad surface. Various ways may be conceivable which are used fordetermining whether the three-dimensional-object-intervening region hasthe correspondence to the road surface. In an example implementation ofthe technology, the determination of whether thethree-dimensional-object-intervening region has the correspondence tothe road surface may be performed on the basis of the density of theblocks that are determined to have the correspondence to the roadsurface.

In this example implementation, when the two three-dimensional objectsare separated from each other and are distant from the own vehicle 1 bythe same relative distance, the road surface determining unit 164 mayidentify the three-dimensional-object-intervening region between the twothree-dimensional objects. For example, the road surface determiningunit 164 may determine a vertical length of thethree-dimensional-object-intervening region on the basis of respectivemaximum lengths of the two three-dimensional objects. Thereafter, theroad surface determining unit 164 may derive a rectangular region thathas the determined vertical length and a horizontal length extendingfrom a right end portion of one of the three-dimensional objects locatedon the left of the image to a left end portion of the otherthree-dimensional object located on the right of the image. The roadsurface determining unit 164 may set the rectangular region to be thethree-dimensional-object-intervening region.

Thereafter, the road surface determining unit 164 may calculate theintegrated value of each block in the identifiedthree-dimensional-object-intervening region.

FIG. 12 illustrates an example process performed by the road surfacedetermining unit 164. For example, a region 240 in the integrated valuemap 224 may be identified as the three-dimensional-object-interveningregion, as illustrated in (a) of FIG. 12. In (a) of FIG. 12, the region240 or the three-dimensional-object-intervening region is enclosed by aheavy line, and may include a total of 24 blocks that are located athorizontal positions 2 to 5 and vertical positions 2 to 7. Theintegrated value of each of the blocks in the region 240 may becalculated by simply adding the determination values of the blocksassigned with “1” (i.e., blocks determined to have the correspondence tothe road surface) in the region 240; however, such addition can takemuch more time for a larger three-dimensional-object-intervening regionincluding a larger number of the blocks. To reduce such a load, in thisexample implementation, the integrated value may be readily calculatedusing the integrated value map 224.

Referring to (b) of FIG. 12, for example, an integrated value of anypredetermined three-dimensional-object-intervening region may becalculated by the following expression:ai−bi−ci+di

where “ai” represents an integrated value of a block located at anupper-right internal part of the predeterminedthree-dimensional-object-intervening region, “bi” represents anintegrated value of a block at an immediate left of a block located atan upper-left internal part of the predeterminedthree-dimensional-object-intervening region, “ci” represents anintegrated value of a block immediately below a block located at alower-right internal part of the predeterminedthree-dimensional-object-intervening region, and “di” represents anintegrated value of a block at an immediate lower left of a blocklocated at a lower-left internal part of the predeterminedthree-dimensional-object-intervening region. Note that, when nocorresponding block exists at a left-end part or a bottom-end part ofthe image, “0” may be assigned to a deemed block.

For example, as illustrated in (a) of FIG. 12, the integrated value ofthe region 240 that includes the blocks located at the horizontalpositions 2 to 5 and the vertical position 2 to 7 is “12”, which iscalculated by the expression 17−5−1+1, where, in order, “17” is theintegrated value of the block located at the upper-right internal partof the predetermined three-dimensional-object-intervening region, “5” isthe integrated value of the block at the immediate left of the blocklocated at the upper-left internal part of the predeterminedthree-dimensional-object-intervening region, “1” is the integrated valueof the block immediately below the block located at the lower-rightinternal part of the predetermined three-dimensional-object-interveningregion, and the “1” is the integrated value of the block at theimmediate lower left of the block located at the lower-left internalpart of the predetermined three-dimensional-object-intervening region.In this way, the integrated value of thethree-dimensional-object-intervening region is readily obtained.

Thereafter, the road surface determining unit 164 may calculate thedensity of the blocks that are determined to have the correspondence tothe road surface (i.e., blocks each assigned with “1”) by dividing theintegrated value of the three-dimensional-object-intervening region bythe area of the three-dimensional-object-intervening region. In theexample illustrated in (a) of FIG. 12, for example, the density is 50%,which is calculated by 12/24.

When the calculated density is not less than a predetermined threshold(e.g., 50%), the road surface determining unit 164 determines that thethree-dimensional-object-intervening region has a correspondence to theroad surface. In contrast, when the calculated density is less than thepredetermined threshold, the road surface determining unit 164determines that the three-dimensional-object-intervening region has nocorrespondence to the road surface. In the example illustrated in (a) ofFIG. 12, for example, the road surface determining unit 164 determinesthat the three-dimensional-object-intervening region has acorrespondence to the road surface, since the density of the blocksdetermined to have the correspondence to the road surface is 50%.

Through such a road surface determining process (S204), whether thethree-dimensional-object-intervening region has the correspondence tothe road surface is properly determined. As a result, thethree-dimensional-object-intervening region between the opposite sideportions 230 b of the rear surface of the preceding vehicle 230 isdetermined to have no correspondence to the road surface in the exampleillustrated in FIG. 11A, whereas thethree-dimensional-object-intervening region between thethree-dimensional objects 232 is determined to have the correspondenceto the road surface in the example illustrated in FIG. 11B. When thethree-dimensional-object-intervening region is determined to have thecorrespondence to the road surface, a subsequent process or thethree-dimensional object composition process (S208) may not beperformed.

[Three-Dimensional Object Composition Process S208]

When the three-dimensional-object-intervening region is determined tohave no correspondence to the road surface, as in the exampleillustrated in FIG. 11A, for example, the three-dimensional objectcomposition unit 166 may regard the three-dimensional objects that areseparated from each other as candidate parts of a unifiedthree-dimensional object.

When the candidate parts satisfy another condition for being athree-dimensional object, the opposite side parts 230 b of the rearsurface of the preceding vehicle 230 are regarded as a unifiedthree-dimensional object, and are eventually identified as a specificobject or the preceding vehicle 230.

As described above, the determination of whether the region betweenthree-dimensional objects that are located at respective positionsdistant from the own vehicle 1 by the same relative distance has acorrespondence to the road surface facilitate proper integration of thethree-dimensional objects. Accordingly, it is possible to achieve properextraction and identification of a three-dimensional object.

According to any implementation of the technology, a program also may beprovided that causes a computer to serve as the vehicle exteriorenvironment recognition apparatus 120, and a computer-readable recordingmedium that stores the program. Non-limiting examples of the recordingmedium may include a flexible disk, a magneto-optical disk, ROM, CD, DVD(Registered Trademark), and BD (Registered Trademark). As used herein,the term “program” may refer to a data processor that is written in anylanguage and any description method.

The central controller 154 illustrated in FIG. 3 is implementable bycircuitry including at least one semiconductor integrated circuit suchas at least one processor (e.g., a central processing unit (CPU)), atleast one application specific integrated circuit (ASIC), and/or atleast one field programmable gate array (FPGA). At least one processoris configurable, by reading instructions from at least one machinereadable non-transitory tangible medium, to perform all or a part offunctions of the central controller 154. Such a medium may take manyforms, including, but not limited to, any type of magnetic medium suchas a hard disk, any type of optical medium such as a CD and a DVD, anytype of semiconductor memory (i.e., semiconductor circuit) such as avolatile memory and a non-volatile memory. The volatile memory mayinclude a DRAM and a SRAM, and the nonvolatile memory may include a ROMand a NVRAM. The ASIC is an integrated circuit (IC) customized toperform, and the FPGA is an integrated circuit designed to be configuredafter manufacturing in order to perform, all or a part of the functionsof the central controller 154 illustrated in FIG. 3.

Although some implementations of the technology have been described inthe foregoing by way of example with reference to the accompanyingdrawings, the technology is by no means limited to the implementationsdescribed above. It should be appreciated that modifications andalterations may be made by persons skilled in the art without departingfrom the scope as defined by the appended claims. The technology isintended to include such modifications and alterations in so far as theyfall within the scope of the appended claims or the equivalents thereof.

For example, in at least one foregoing example implementation, thedetermination values “1” and “0” that are obtained through thedetermination of correspondence to the road surface may be simplyintegrated by the three-dimensional object identifying unit 162;however, in consideration of a case where the correspondence to the roadsurface is not clearly determined, reliability values that indicate alikelihood of correspondence to the road surface may be determined perblock, in an alternative example implementation of the technology. Inthe alternative example implementation, the road surface identifyingunit 160 may determine the reliability values (indicated in a range from0 to 1) per block, and calculate an integrated value, on a block basis,along both or one of the vertical direction and the horizontaldirection. This allows for a more definite determination of whether thethree-dimensional-object-intervening region has the correspondence tothe road surface.

In at least one foregoing example implementation, the three-dimensionalobject identifying unit 162 may perform integration of the determinationvalues obtained through the determination of correspondence to the roadsurface or integration of the reliability values that indicate thelikelihood of correspondence to the road surface, on a block basis,along both or one of the vertical direction and the horizontaldirection. Alternatively, various calculations, such as multiplication,that make it possible to calculate the density of the blocks determinedto have the correspondence to the road surface may be used, instead ofthe integration (or addition).

Further, in at least one foregoing example implementation, theintegration of the determination values of the blocks determined to havethe correspondence to the road surface or the integration of thereliability values indicating the likelihood of correspondence to theroad surface may be performed from the lower-left part of the image inthe vertically upward direction or the horizontally rightward direction.However, the integration may be performed along another direction. Inanother example implementation of the technology, the integration may beperformed along both or one of the vertical position and the horizontaldirection, from an upper left part, an upper right part, or a lowerright part of the image to the direction in which the image exists.

A part or all of the processes in the vehicle exterior environmentrecognition procedure as disclosed herein does not necessarily have tobe processed on a time-series basis in the order described in theexample flowchart. A part or all of the processes in the vehicleexterior environment recognition procedure may involve parallelprocessing or processing based on subroutine.

The invention claimed is:
 1. A vehicle exterior environment recognitionapparatus configured to recognize an environment outside an own vehicle,the apparatus comprising: a road surface identifying unit configured toidentify a road surface in an image; a three-dimensional objectidentifying unit configured to identify three-dimensional objects eachhaving a height extending vertically upward from the identified roadsurface; a road surface determining unit configured to perform adetermination of whether a three-dimensional-object-intervening regionbetween the three-dimensional objects identified by thethree-dimensional object identifying unit has a correspondence to theroad surface, when the identified three-dimensional objects areseparated from each other and are located at respective positionsdistant from the own vehicle by a same relative distance; and athree-dimensional object composition unit configured to regard theidentified three-dimensional objects that are separated from each otheras candidate parts of a unified three-dimensional object, when thethree-dimensional-object-intervening region is determined to have nocorrespondence to the road surface by the road surface determining unit,wherein the road surface determining unit performs the determination ofwhether the three-dimensional-object-intervening region has thecorrespondence to the road surface based on blocks in thethree-dimensional-object-intervening region located within a relativedistance from the road surface.
 2. The vehicle exterior environmentrecognition apparatus according to claim 1, wherein thethree-dimensional object identifying unit determines determinationvalues per block that includes a plurality of pixels, and performsintegration of the determination values, on a block basis, along both orone of a vertical direction and a horizontal direction, thedetermination values indicating the correspondence to the road surface,and the road surface determining unit performs the determination ofwhether the three-dimensional-object-intervening region has thecorrespondence to the road surface, on a basis of an integrated value ofthe blocks in the three-dimensional-object-intervening region.
 3. Thevehicle exterior environment recognition apparatus according to claim 1,wherein the three-dimensional object identifying unit determinesreliability values per block that includes a plurality of pixels, andperforms integration of the reliability values, on the block basis,along both or one of a vertical direction and a horizontal direction,the reliability values each indicating a likelihood of thecorrespondence to the road surface, and the road surface determiningunit performs the determination of whether thethree-dimensional-object-intervening region has the correspondence tothe road surface, on a basis of an integrated value of the blocks in thethree-dimensional-object-intervening region.
 4. A vehicle exteriorenvironment recognition apparatus configured to recognize an environmentoutside an own vehicle, the apparatus comprising circuitry configured toidentify a road surface in an image, identify three-dimensional objectseach having a height extending vertically upward from the identifiedroad surface, perform a determination of whether athree-dimensional-object-intervening region between the identifiedthree-dimensional objects has a correspondence to the road surface, whenthe identified three-dimensional objects are separated from each otherand are located at respective positions distant from the own vehicle bya same relative distance, and regard the identified three-dimensionalobjects that are separated from each other as candidate parts of aunified three-dimensional object, when thethree-dimensional-object-intervening region is determined to have nocorrespondence to the road surface, wherein the determination of whetherthe three-dimensional-object-intervening region has the correspondenceto the road surface is performed based on blocks in thethree-dimensional-object-intervening region located within a relativedistance from the road surface.